Conventional traveling matte composite photography techniques are known and have become a routine part of motion picture and television post production. These processes are described in, among others, U.S. Pat. Nos. 4,100,569; 4,344,085; 4,625,231; 5,032,901; 5,343,252; 5,424,781; 5,742,354; 6,134,345; and 6,288,703. For the purpose of this document, a matte is regarded as a set of numerical values, one for each pixel of an image, which specifies which pixels are considered opaque (i.e., value of 1.0) and which pixels are considered transparent (i.e., value of 0.0), with “transitional” values (i.e., the edges of the opaque regions) possibly having a value between 0.0 and 1.0.
The Color Difference Traveling Matte System is the most popular and flexible of the single film compositing techniques. It can be used with standard cameras, any color film, and it does not require the use of filters. The only special requirement is that the background and floors must be painted blue and illuminated evenly. The Color Difference Traveling Matte System is based on the colorimetry of colors as follows: excepting the colors blue and magenta, all colors have a blue content that is equal to, or less than, their green content. All the remaining colors except yellow and green have equal blue and green content. When blue and green are equal, their B & W separations will be identical. Thus, there is no need to make a blue separation to reproduce such colors as reds, flesh tones, all shades of pink, white, gray and all saturations of cyan. Since the blue and green separations (for these specific colors) are identical, one would simply use the green separations twice; once as the green printing separation, and once as the blue printing separation.
The traditional optical techniques described above have given way to newer electronic and digital methods of compositing. While these new electronic methods replace optical printing and the use of film mattes, they nevertheless subscribe to the same color difference theory of the traditional techniques. For example, a computer simply removes everything that is a particular color. This allows the remaining objects to have a substitute background inserted electronically behind them. While this explanation may be simplistic with respect to certain systems, it is nevertheless accurate. One problem that may result from using color as a basis for image compositing is that an abrupt difference from the object and a background sometimes results in hard edges in the final composite. This problem has been greatly reduced, however, with software improvements. The most noticeable aspect of a composited image is the lighting differences between the objects recorded on a blue screen and the backgrounds they are composited into. Once again this problem can be greatly reduced by careful and time-consuming lighting of the objects being composited so they match their final backgrounds.
Many image compositing systems have been developed that provide good results. Special effects are now commonplace in even modest budget films. While image compositing is routine, it still requires painstaking frame-by-frame image “correction” on the part of an operator or additional shoot days on special stages designed for recording images for subsequent compositing. These additional production steps are both time-consuming and expensive.
As more and more films deal with the fantastic, specific periods in time, or just your run of the mill destruction of a city, images recorded on location are increasingly being married with those created in a computer or shot as miniatures. The demand for image compositing is ever increasing and is becoming a significant line item in the overall film budget. The advent of digital film scanning, electronic image manipulation, and computer-generated imagery has created a post production infrastructure with unprecedented power over the moving image. Advances in computing speed, software algorithms, and common digital file transfer protocols have all been developed to service the ever-increasing demand for image compositing.
A new approach to image compositing exploits the ability to displace foreground and background objects from the subject using a moving point of view pivoted or converged on the subject of the scene being recorded. Parallax scanning lenses and square-wave camera arrays are two devices useful for creating foreground and background displacement. Co-assigned U.S. Pat. Nos. 4,815,819; 4,966,436; 5,014,126; 5,157,484; 5,325,193; 5,444,479; 5,448,322; 5,510,831; 5,678,089; 5,699,112; 5,933,664; 5,991,551; and 6,324,347, which are hereby incorporated by reference, teach methods and means for square-wave, slit scanning, and parallax scanning.
In all of the above referenced patents, the greater the angle of parallax difference in the captured point of view, the greater the amount of foreground and background displacement. While noticeable image instability is an undesirable trait in normal image capture (filming), it can be useful in image compositing. For example, an image captured in the method described in U.S. Pat. No. 5,448,322 would have acceptable stability when filmed with a parallax scan angle of 0.035° at a frequency of 4.3 Hz. However, the same image would become unacceptable if the parallax scan angle were increased to 0.1° at 4.3 Hz. This is because objects in front of and behind the plane of focus would move in a circular motion relative to one another. Nevertheless, objects at or near the plane of focus would remain still, regardless of the foreground and background motion. This is because the optical axis of the moving optical element (MOE) lens pivots on the center of the plane of focus when parallax scanning, much the same as the support for a playground teeter-totter remains fixed while both ends are free to move up and down.
The present inventors have spent considerable time developing a moving image (film, video, or high definition) lens system that will produce stable depth enhanced images and have concluded that unstable images can be useful as well. One of the great difficulties in image processing is edge detection, in which a computer must determine where the edges of a particular object in a scene start and stop. The traditional methods discussed above have made this determination based on color.
It is possible, however, to use motion induced by a parallax scan or other means to determine where to “clip” objects from the background. When everything is moving in a regular pattern at some constant frequency with regard to a convergence point in the scene being captured, then objects at or near the point of convergence can be identified and located. The goal is to move the point of view in a manner that is unlikely to be encountered in nature or in the action of the scene being recorded. And if a similar motion is present in the recorded scene, the frequency and direction of the moving point of view can be changed. In addition, the clipping can be adjusted to include a range of objects and talent.
A parallax scan-based compositing process has several advantages. Objects and/or talent can be recorded on location as an additional pass (take) during principal photography. A number of blue screen shoot days may be reduced or eliminated. Objects and talent lighting will match the composited location backgrounds, and the technology can be applied to broadcast and consumer video applications.
The present invention solves one or more of the problems associated with known image compositing processes.